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Ss PT5 Background 


¢ Motivation 
— Build upon previous human-in-the-loop simulations results and lessons learned to 
identify minimum DAA display and guidance requirements for draft SC228 MOPS 
e PT4 
— A suite of integrated guidance tools led to faster pilot responses, fewer losses of 
well clear and less severe losses of well clear when they did occur 
¢ iHITL 
— Integrated guidance tools led to less severe losses of well clear and faster pilot 
responses than seen in PT4 
e AFRL Maneuver Study 


— Guidance (in the form of “‘banding’) led to faster pilots responses and fewer 
collision avoidance alerts 
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PT5 Background 


¢ Modifications from previous sims: 


Guidance tools were no longer tightly coupled to the ground control station’s auto 
pilot interface 


Removed advanced features present in iHITL (e.g., well clear ring & dead reckoning 
lines) 
Modeled sensor uncertainty for the first time 

¢ Critical to test displays and algorithms with ‘imperfect’ data prior to flight tests 
Implemented alerting structure as part of the draft MOPS 
Increased workload on the pilot 


¢ More secondary tasks and interaction with their route 
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Experimental Design 


¢ Mixed Factorial Design 
— Display Configuration (Within-Subjects Independent Variable): 


Configuration 1: Minimum Information Set (No Guidance) 
Configuration 2: Stratway+ No Fly Bands 

Configuration 3: JADEM Omni Bands 

Configuration 4: JADEM Vector Planning Tools 





Ss Experimental Design 


Participants 


— 16 active UAS pilots 

¢ Avg. 37 years old (all male) 

¢ Manned Flying Experience 
— Civil airspace: 575 avg. hrs 
— Military: 1760 avg. hrs 

¢ Unmanned Flying Experience 
— Civil airspace: 30 avg. hrs 
— Military: 1100 avg. hrs 





Task 
— Fly simulated MQ-9 through Class E airspace (Oakland Center — ZOA 40/41) 
¢ Navigate along pre-filed routes (used AFRL's Vigilant Spirit Control Station) 
¢ Maintain well clear from pre-scripted conflicts 
¢ Coordinate with ATC 
e Attend to secondary tasks (e.g., chat messages, system alerts) 





Ss Experimental Design 


¢ Pre-planned conflicts with ownship 
— 6scripted encounters predicted to lose well clear 
¢ 3 with cooperative traffic (detected at max range of 15nm) 
¢ 3 with non-cooperative traffic (detected at max range of 8nm, with limited FoR) 


— 3scripted encounters predicted to become preventive self separation alerts 





Ss Experimental Design 


¢ Simulation confederates 
— NATCA controller managed UAS and manned traffic within ZOA 40/41 
¢ Simulated manned traffic based on actual sector activity 
— Pseudo-pilots managed all manned traffic to provide dynamic sector activity 
— ATCSME operated as ‘ghost’ controller to ensure conflicts were generated 





Air Traffic Control Station (MACS) Pseudo Pilot Station 


Draft MOPS Alerting Structure 


. Buffered Well Time to Loss of Well | Aural Alert 
Symbol Name Pilot Action Ae 
Clear Criteria Clear Verbiage 


DMOD = 0.75 nmi 
HMD = 0.75 nmi 
ZTHR = 450 ft 
modTau = 35 sec 
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“Traffic, 
Maneuver 
Now” 


25 sec 
(TCPA approximate: 
60 sec) 


Immediate action required 
Notify ATC as soon as practicable after 
taking action 


Self Separation 
Warning Alert 


75 sec 
(TCPA approximate: 
110 sec) 


On current course, corrective action DMOD = 0.75 nmi 

Corrective Self required HMD =0.75 nmi 
Separation Alert Coordinate with ATC to determine an ZTHR = 450 ft 

appropriate maneuver modTau = 35 sec 


“Traffic, 
Separate” 


On current course, corrective action DMOD = 0.75 nmi 
Preventive Self should not be required HMD = 1.0 nmi 
Separation Alert Monitor for intruder course changes ZTHR = 700 ft 
Talk with ATC if desired modTau = 35 sec 


75 sec 
(TCPA approximate: 
110 sec) 


“Traffic, 
Monitor” 





DMOD = 0.75 nmi 
Self Separation Monitor target for potential increase in HMD = 1.5 nmi 
Proximate Alert threat level ZTHR = 1200 ft 
modTau = 35s 


85 sec 
(TCPA approximate: N/A 
120 sec) 





Within surveillance 


None (Target) No action expected field of regard 
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Display Conditions 
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Ss Encounter Statistics 


Total Number of Encounters: 

— 282 encounters analyzed in following measured response data 

¢ 37 (13%) encounters were excluded due to pilot maneuvering prior to receiving an alert 

By Threat Level 

— 244 Corrective SS Alerts issued 

— 111 SS Warning Alerts issued 
By Intruder Equipage 

— 138 encounters with Cooperative Traffic 

— 144 encounters with Non-Cooperative Traffic 
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Measured Response Timeline & Associated Metrics 








Total 
BHRBR BREE HEBER BE BBR RR BRR RRR RRR RRR eR eR Ree eee ee eee . SEB, 
Response Time = 
V V 
Initial Initial Edit 
: BEE BBP Ree Response Time BEE BEB : SEE eee Time See e8 : 
y vy (First Upload) vy 
... Notification ,,,., panties Total Edit Time ea aeny 
° Time ° r (Final Upload) : 
V V V V 
Traffic Display Pilot Notifies ATC Approval Pilot Initiates Pilot Uploads Pilot Uploads 
Alert (SS or CA) ATC Edit First Edit Final Edit 
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Ss Notify Time 
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Configuration 


¢ There was no significant effect of Configuration on Notify Time for all SS alerts, 
p> .05 
¢ On average, pilots took 16.15s to notify ATC in response to a Corrective SS or SS 
Warning alert 
— 50% within 12s, 90% within 33s 





al Initial Response Time 
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¢ Configuration had a significant effect on Initial RTs, p < .01: 
— Initial RTs for Stratway+ were 7.03s shorter, on average, than those in Vector Planner, p < 0.5 
— Initial RTs for Omni Bands were 5.07s shorter, on average, than those in Vector Planner, p = .05 
— No other configurations differed significantly 
° On average, pilots took 12.35s to initiate an edit in response to a Corrective SS or SS 
Warning alert 
— 50% within 12s, 90% within 36s 
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Initial Edit Time 
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Info Only Stratway+ Omni Bands Vector Planner 
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Info Only 





Stratway+ Omni Bands’ Vector Planner 


Configuration 


No effect of configuration on initial edit times (p > .05) 
On average, pilots took 8s to make an initial edit following a Corrective SS or 


SS Warning alert 
— 50% within 6s, 90% within 16s 


Total Edit Time 
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¢ Configuration had a significant effect on Total Edit times, p < .05: 


— Total Edit times for Vector Planner were 6.6s shorter, on average, than those in 
Info Only, a significant difference, p < 0.1 


— No other configurations differed significantly 
¢ On average, pilots took 10.7s to upload their final maneuver after the 
initiation of an edit 
— 50% within 7s, 90% within 22s 


Ss Total Response Time 
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¢ Configuration had a significant effect on Total RTs: 
— Total RTs for Stratway+ were 8.2s shorter, on average, than those in Info Only, a 
Significant difference (p=0.02) 
— No other configurations differed significantly 
° On average, pilots took 22.8s to upload a final maneuver following the onset 
of a Corrective SS or SS Warning alert 
— 50% within 19s, 90% within 42s 
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Maneuver Types 








Maneuver Type by Display 


80 


@ Lateral 






61 WBoth 





5 Info Only 66 (85%) 7 (9%) 5 (6%) 
S$ Stratway+ 64 (88%) 8 (11%) 1 (1%) 73 
O 
9 Omni Bands 61 (92%) 4 (6%) 1 (2%) 66 
3 
Vector 58 (89%) 4 (6%) 3 (5%) 65 
Planner 
Total 249 (88%) 23 (8%) 10 (4%) 282 


Info Only Stratway+ Omni Bands Vertical Planner 


Configuration 


¢ Maneuver Type 
— 88% of maneuvers were in horizontal dimension 
— 8% of maneuvers were in vertical dimension 
— 4% of maneuvers were made in both dimensions 
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Ss Horizontal Maneuver Size 


60 


45.67 





Info Only Stratway+ Omni Bands Vector Planner 


Configuration 


Info Only resulted in horizontal maneuvers that were, on average, 10deg larger 
than seen with the three guidance displays 


— Also see less variability between the three guidance displays 


Not enough data points to look at average vertical maneuver size 
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Ss Number of Uploads 


% Encounters Requiring Multiple Uploads 
100% 
90% 
80% 
70% 
60% 
50% 


Percentage 


40% 
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10% | 
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Info Only Stratway+ Omni Bands Vector Planner 






12.31% 


8.22% 4.55% 





Configuration 


e Pilots required an average of 1.34 uploads per encounter in Info Only 
condition, compared to (roughly) 1.10 uploads per encounter with the 
remaining displays 

— 25% of encounters in the Info Only display required more than 1 upload to 
maintain separation 
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Ss Results Summary 


¢ Notification Time & Initial Edit Time 
— No differences 
¢ Initial Response Time 
— Vector Planning tools resulted in the slowest times, while remaining 3 displays were 
roughly equal 
¢ Total Edit Time 


— Vector Planning tools resulted in fastest times, with banding displays slightly slower and 
Information Only the slowest 


¢ Total Response Time 


— Fastest for Stratway+, but very close with Omni Bands 
¢ Vector Planning tools substantially slower, but Information Only far slower 


e Maneuver Type 

— Overwhelming preference for lateral maneuvers, little variation between displays 
¢ Maneuver Size 

— Larger lateral maneuvers for Information Only display, little difference between rest 
¢ Encounters with Multiple Uploads 


— Least common with 2 banding displays 
¢ Vector planner roughly twice as likely, information only 4x as likely 
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Ss Conclusion 


¢ Suggestive maneuver guidance resulted in faster responses and more efficient 
maneuvers 
— The 2 banding displays (Stratway+ & Omni Bands) helped maintain consistently low 
interaction times — initial RT, initial & total edit 
¢ Vector Planner increased edit times but raised initial RT times 
— The 2 banding displays also minimized maneuver size and the number of multiple- 
upload maneuvers 


¢ Vector planner helped minimize maneuver sizes but led to twice the number of multiple- 
upload maneuvers than the banding displays 
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